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ABSTRACT 
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WO PCT/USOO/25089 1/2001 



An optical fiber ring network includes a plurality of inter- 
connected nodes, each pair of neighboring nodes being 
interconnected by a pair of optical links. Using coarse 
wavelength division multiplexing, data is transmitted in both 
directions over each link, using a first wavelength XI to 
transmit data in a first direction over the link and a second 
wavelength X2 to transmit data in a second, opposite direc- 
tion over the link. The two wavelengths XI and X2 differ by 
at least 10 nm. Each of the data streams transmitted over the 
optical link has a bandwidth of at least 25 Gbps. Further, 
each data stream has at least two logical streams embedded 
therein. A link multiplexer at each node of the network 
includes one or more link cards for coupling the link 
multiplexer to client devices, and one or more multiplexer 
units for coupling the link multiplexer to the optical links. 
Each link card includes frame buffers capable of storing 
numerous Fibre Channel frames that are being transmitted to 
and from the client device(s) coupled to that link card. The 
link card also includes flow control logic for pre-filling the 
frame buffers with frames of data before the receiving client 
devices send flow control messages to request their trans- 
mission. The combined effect of the frame buffers and flow 
control logic is that the full bandwidth of the links can be 
utilized even when the network nodes are very far apart and 
the client devices have small input data buffers, 

21 Claims, 14 Drawing Sheets 
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OPTICAL FIBER RING COMMUNICATION FIG. 2 is a block diagram showing multiple physical 

SYSTEM communication paths between network nodes; 

FIG. 3 is a block diagram of a link multiplexer, for use at 

This application claims the benefit of Provisional appli- any one node of the fiber optic ring network; 

cation Ser. No. 60A53,519, filed Sep. 13, 1999. 5 na 4 ^ a block of a link card> which ^ a 

The present invention relates generally to optical fiber component of the link multiplexer of FIG. 3; 

communication systems, and particularly to a system archi- , , ., , , ,. i- * , 

. _ r i * cc • * c *• i cu FIG. 5 is a detailed block diagram of a link card: 

lecture for making efficient use of optical fiber communi- . & 9 

cation rings and provided reliable logical connections FIG * 6 fe a block diagram of a Mu * Interface Frame 

between network nodes. 10 Processor, which is a component of the link card of FIG. 5; 

FIG. 7 is a block diagram of a fink card FC Link Interface 

BACKGROUND OF THE INVENTION Frame Processor, which is a component of the link card of 

Fiber optic rings have been installed, and as of 1999 are . . , . r , , „^ T . , T 

in the process of being installed in many cities. These FIG. 8 is a block diagram of a link card GE Link Interface 

communication networks have the potential to provide low 15 P"> ces s<*> is a component of the link card of 

cost, high bandwidth connections within a geographical area ^» 

of several miles, as well as low cost, high bandwidth FIG. 9 is a block diagram of a multiplexer unit, which is 

connections to other communication networks, such as the a component of the link multiplexer of FIG. 3; 

Internet. To date, however, the equipment available for FIG. 10 is a block diagram of a time division multiplexer 

providing data communications over these networks has and transmission rate smoothing circuit, which is a compo- 

been sufficiently expensive that relatively little use is being nent of the multiplexer unit of FIG. 6. 

made of these fiber optic networks. piG. U is a block diagram of a receive datapath circuit, 

Tne present invention provides a low cost system archi- multiple instances of which are used in the time division 

tecture that allows Fibre Channel (FC) and Gigabit Ethernet 25 multiplexer and transmission rate smoothing circuit of FIG. 

(GE) data streams to be seamlessly routed through such fiber 7; 

optic ring networks with very high reliability, all while FIG. 12 is a block diagram illustrating a segment of a fiber 

making efficient use of the available bandwidth. 0 ptj c rm g network; 

SUMMARY OF THE INVENTION FIG * 13 is a block illustrating a fiber optic ring 

30 network in accordance with the present invention; 

An optical fiber ring network includes a plurality of FIG. 14 is a block diagram illustrating how the fiber optic 

interconnected nodes, each pair of neighboring nodes being ring network illustrated in FIG. 13 is reconfigured during 

interconnected by a pair of optical fiber links. Using coarse failover caused by a broken fiber; 

wavelength division multiplexing, data is transmitted in both FIG. 15 is a block diagram illustrating how the fiber optic 

directions over each optical link, using a first optical wave- 35 ^ network illustrated m FIG. 13 is reconfigured during 

length XI to transmit data in a first direction over the link and f ai i over cause d by a failed (client device) node; and 

a second optical wavelength X2 to transmit data in a second, mQ lfi fa a blocR d{ iUustratmg how the fiber optic 

opposite direcUon over the hnk. The two optical wave- network iUasMM ^ FIG 13 ^ recoDfigured during 

lengths XI and X2 differ by at least 10 am. f J mr caused by a fafled ^ at a head ^ of ^ m * 

In a preferred embodiment, each of the data streams 0 p^ c rm g network 
transmitted over the optical link has a bandwidth of at least 

2.5 Gbps. Further, each data stream has at least two logical DESCRIPTION OF THE PREFERRED 

streams embedded therein. EMBODIMENTS 

At each node of the network there is a link multiplexer 45 Referring to FIG. 1, there is shown a fiber optic ring 

that includes one or more hnk cards for coupling the link network 100. The network includes a pair of fiber optic 

multiplexer to client devices, and one or more multiplexer caD i es 102 that traverse a loop or ring. At each node 104 of 

units for coupling the link multiplexer to the optical links. me network, the fiber optic cables are segmented so that 

Each link card includes frame buffers capable of storing signals on the pair of fiber optic cables are received by the 

numerous Fibre Channel frames that are being transmitted to 5Q ^ multiplexer 106 at that node, and then either processed 

and from the client device(s) coupled to that link card. The 0 r forwarded to the next segment of the fiber 102. 

fink card also includes flow control logic for pre-filling the ^ e link multiplexers 106 perform numerous functions: 

frame buffers with frames of data before the receiving client forwarding signals from one opucal fibcr segment to the 

devices send flow control messages to request their trans- next> routm si k from the dcal fiber caWes to clieQt 

mission Hie combined effect of the frame buffers and flow s$ dcvkcs 0f rommunication Uncs , and rollliag s ^ nals to thc 

control logic is that the full bandwidth of the links can be tical fiber cables from cUeQt devices Qr communication 

utilized even when the network nodes are very far apart and ^ Thc ^ multiplcxers i 06 also combine signals from 

the chent devices have small input data buffers. mu]tiple mms ^ time divfcion Md wave]englh divi . 

BRIEF DESCRIPTION OF THE DRAWINGS s * on tecnmc I ues so as to transmit them over the fiber optic 

60 cables 102. 

Additional objects and features of the invention will be The link multiplexer 106 at each node 104 is typically 

more readily apparent from the following detailed descrip- coupled to other devices or communication lines via a 

tion and appended claims when taken in conjunction with switch or switch fabric 108. The switch 108 connects 

the drawings, in which: various devices and communication channels to the host (or 

FIG. 1 is a block diagram a fiber optic ring network 65 client) side ports of the link multiplexer. The switches 108 

having a plurality of nodes that employ the present inven- are generally conventional switching devices and fabrics, 

tion; such as time division multiplexed busses, and in some 
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3 

circumstances are not needed, and are therefore not 
described further in this document. 

An example of a node 104-1 is a node that is coupled to 
a local area network (LAN). The LAN may, in turn, be 
coupled to any number of server computers 110 and end user 
workstations 112. The LAN may be coupled to link multi- 
plexer 106 for the node 104 by a switch 108 and router 114, 
or perhaps just a router 114 if no other switching function- 
ality is required. 

Asecond example of a node 104-2 is one that provides an 30 As shown in FIG. 2, port Aof Mux 1 in one node is always 

Internet connection 116 to the network 100, via a router 114 connected to port B of Mux 1 in a neighboring node, and 

that is coupled to the link multiplexer 104-2. similarly port A of Mux 2 in one node is always connected 

Another example of a node 104-3 is one that contains a <° P ort * Mux 2 m ^ neighboring node As a result, the 

"disk farm" 118, which is generally a set of disks for „ ^ £ ow m one ^ectxon through the ring whde the 

providing centralized data storage used by devices coupled 15 U flow m ** °PP 0Slte 

to other ones of the nodes 104. The present invention makes In another embodiment, four data signal signals are trans- 
it practical for companies with buildings at multiple loca- mitted over each optical fiber cable segment, using four 
tions throughout a city or similar region to use centralized distinct transmission optical wavelengths, each separated 
data storage. The speed of data access provided by the fiber „ & om ^ c others by at least 10 nm, and preferably separated 
optic network 100 is so high that there is little if any &om each omer by 20 nm or more. In mis embodiment, each 
perceptible difference to end users between having data link multiplexer uses four Mux units 142. In one implemen- 
storage in each of the facilities as compared with having data tation of ^ embodiment, the four optical wavelengths used 
storage at a single, central storage node on the ring network. are 15 1° nm > 1530 nm, 1550 and 1570 nm. 

_ In yet another embodiment, each data signal transmitted 

Multiple Physical Paths Between Each Pair of over ^ optical fiber cables transmits data at a speed of 5.0 

Nodes Gbps or 10 Gbps, thereby doubling or quadrupling the 

Referring to FIG. 2, in a preferred embodiment, the link bandwidth of the optical fiber ring, without increasing the 

multiplexers 104 (FIG. 1) contain optical transceivers that number of optical fiber cables used. 

both transmit and receive signals on each optical fiber 102. 30 T - 1 n- i a u * 

A 1. u . T^l . i_ * 1 Link Multiplexer Architecture 

As a result, when two optical fibers are used with a physical r 

transmit and receive channel on each fiber, there are four Referring to FIG. 3, there is shown a preferred embodi- 
bidirectional communication paths between any two nodes ment of a link multiplexer 106. The link multiplexer 106 
of the network, so long as all the optical fiber cable segments includes up to eight link cards 140, and two multiplexer 
are connected and intact. For example, the bidirectional 35 units 142. Each link card 140 provides a high speed con- 
physical signal paths between Node 1 and Node 2 are as nection to a client device or communication channel. In a 
follows: 1) OL12-1, 2) OL12-2, 3) OL13-1-OL23-1, and 4) preferred embodiment two types of link cards are used: one 
OL13-2-OL23-2. Since there are four full bandwidth physi- for Fibre Channel connections, which operate at 1.0625 
cal signal paths between each pair of nodes in the network, Gbps, and another for connections to Gigabit Ethernet 
the impact of any single segment failure is much less severe 40 channels, which operate at 1.25 Gbps. Internally, the link 
than in prior art systems where there are just two physical cards operate at 1.25 Gbps. In an alternate embodiment, the 
signal paths between each pair of nodes. In such prior art link cards and multiplexer units operate at a rate that is 
systems, each optical fiber channel is used to send signals in greater than 1.25 Gbps, such as 1.28 Gbps or 1.5 Gbps or 
only one direction. even 2.0 Gbps, Additional types of link cards can be used to 
The bidirectional signal paths on each optical fiber are 45 transmit and receive signals using other signal protocols, so 
formed using a particular type of "coarse wavelength divi- lon S ^ & e maximum data rate of the signals does not exceed 
sion multiplexing." In particular, one optical wavelength is toe channel bandwidth of the link multiplexer (which is 1.25 
used to transmit a stream of data, while a second optical Gb P s in ^e preferred embodiment). The link cards 140 
wavelength is used to receive a stream of data. Using included in a link multiplexer 106 can include any combi- 
"coarse" wavelength division multiplexing means that the 50 nation of Fibre Channel and Gigabit Ethernet link cards 
optical wavelengths of the two optical signals are at least 10 ( e -8-> two FC link cards > or two GE link cards > or oae of 
nm apart (and preferably at least 20 nm apart) from each cacn * or four °f eaco > ^ s0 on )- 

other. In a preferred embodiment, each optical fiber cable Each multiplexer unit 142 handles up to four full duplex, 

carries a first 25 Gbps data stream at 1510 nm, and a second full bandwidth Fibre Channel (FC) or Gigabit Ethernet (GE) 

2.5 Gbps data stream at 1570 nm. The two data streams flow 55 data streams. More specifically, each multiplexer can trans- 

in opposite directions. Since two optical fiber cables are mit as much as 5.0 Gbps of data, over two physical channels 

coupled to each link multiplexer, there are actually four that each operate at 2.5 Gbps, and can receive as much as 5.0 

optical fibers (i.e., four cable segments) that are coupled to Gbps of data, over two other physical channels that each 

the link multiplexer: two that are coupled to a first neigh- operate at 2.5 Gbps. 

boring node on the ring and two that are coupled to a second 60 

neighboring node on the ring. For simplicity and ease of Card 

explanation, it will be assumed here that data flowing Referring to FIG. 4, there is shown a functional block 

"clockwise" through the ring is transmitted at a first optical diagram of a link card 140. In particular, the link card 140 

wavelength XI (e.g., 1510 nm) and that data flowing "coun- includes a Fibre Channel or Gigabit Ethernet interface 150 

terclockwise" through the ring is transmitted at a second 65 for coupling to a full duplex Fibre Channel or Gigabit 

optical wavelength >l2 (e.g., 1570 nm). This is shown more Ethernet data stream. The interface 150 can also be used to 

specifically in the following table: couple to fractional data streams, in particular to half band- 
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width data streams (operating at 0.503125 Gbps for Fiber 
Channel or 05125 Gbps for Gigabit Ethernet). 

Two buffers 152 are used to buffer data being transmitted 
in each direction, in particular for providing retiming 
between the clock domains of the Mux units (i.e., Mux 1 and 
Mux 2) and the clock domains of client device(s). Finally, a 
Mux unit interface 154 is used to couple the link card to the 
two multiplexer units 142-1 and 142-2. 

FIG. 5 shows a more detailed diagram of the link card 
140, which will now be described starting with gigabit 
interface cards (GBICs) 160 at the bottom of FIG. 5 and 
working upwards toward the Mux unit interface 154. The 
link card 140 includes a pair of GBICs 160, each of which 
couples the link card 140 to a full duplex Fiber Channel or 
Gigabit Ethernet transmission media 158. The transmission 
media, which connects the client devices 159 (also some- 
times called the host system or client communication 
channel) to the link card 140, is typically a coaxial cable or 
fiber optic cable. The GBICs 160 transmit and receive serial 
data streams. 

In order to describe the data flow in a consistent manner, 
the data stream from the Mux units 142 to the client devices 
is referred to as an outbound data stream, and the data stream 
in the opposite direction is referred to as an inbound data 
stream. 

Each of the two GBICs 160 is coupled to a respective 
link interface frame processor 164 by a respective serialize r/ 
deserializer (SERDES) circuit 162, such as the Vitesse 
VSC7125 (for Fibre Channel data streams running at 1.0625 
Gbps) or the Vitesse VSC7135 (for Gigabit Ethernet data 
streams running at 1.25 Gbps). The SERDES 162 converts 
the inbound serial data stream received from the GBIC 160 
into a 10-bit parallel data stream and transmits the converted 
data stream to the link interface frame processor 164. Also, 
the SERDES 162 converts a 10-bit parallel outbound data 
stream received from the link interface frame processor 164 
into a serial data stream and transmits the converted data 
stream to the GBIC 160. 

The link interface frame processor 164 decodes 10ft 
symbols in the inbound data stream from the GBIC into 8ft 
symbols, and encodes 8b symbols received from an out- 
bound frame buffers 168 into 10ft symbols suitable for 
transmission. The link interface frame processor 164 also 
controls the operation of an inbound frame buffer 166 and 
the outbound frame buffer 168. 

A link card channel, which includes the GBIC 160, 
SERDES 162, link interface frame processor 164, and a pair 
of inbound and outbound FIFO (first-in-first-out) frame 
buffers 166,168, can operate in one of two modes under user 
control: distance buffering enabled or disabled. When dis- 
tance buffering is disabled the data frames and flow control 
primitives are passed through the inbound and outbound 
frame buffers 166, 168 as quickly as possible. Whenever 
there is a data frame or flow control primitive in the 
outbound buffer 168, it is read from the buffer 168 and 
passed to the client device immediately through the link card 
channel. All data frames and flow control primitives 
received from the client device are written to the inbound 
frame buffer 166. 

When distance buffering is enabled the link interface 
frame processor 164 receives and interprets flow control 
primitives received from the client device and then controls 
reading the data from the outbound frame buffer 168 as 
requested by the client device. The client device controls the 
reading of data from the outbound frame buffer 168 using 
flow control primitives. The flow control primitives are not 
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passed through the buffers 166, 168 when distance buffering 
is enabled. Instead, the flow control primitives are consumed 
by the link interface frame processor 164. Similarly, when 
distance buffering is enabled, the link interface frame pro- 

5 cessor 164 generates flow control primitives to send to the 
client device based on the fullness of the inbound frame 
buffer 166. Furthermore, when distance buffering is enabled 
the Mux interface frame processor 170 generates and 
receives it's own flow control primitives that are sent to the 

10 link card(s) connected thereto. It should be noted that the 
buffers 166, 168 preferably do not overflow in normal 
operation with distance buffering either enabled or disabled. 

The link interface frame processor 164 extracts "extra** 
Idle symbols (sometimes called "Idles'*) from the outbound 

* 5 data stream, storing only data frames and one Idle between 
data frames in the inbound FIFO frame buffers 166. Thus, if 
there are more than one Idle between data frames the extra 
ones are not stored in the inbound FIFO frame buffer 166. 
For the outbound data streams, the link interface processor 

20 164 inserts as many Idles as may be needed to fill the space 
between data frames being transmitted to the client devices. 

When the link interface frame processor 164 reads a link 
synchronization character from the outbound frame buffer 
168, and the outbound frame buffer 168 is otherwise empty, 

25 link synchronization words are replicated until other data is 
received and stored in the outbound frame buffer 168. Each 
word stored in the frame buffers 166 and 168 includes a pair 
of 8-bit characters, a flag to indicate if the first character of 
the pair is a "K" character, and a parity bit, for a total of 18 

30 bits. K characters are special symbols used for control, and 
thus are not ordinary data. Examples of K characters are 
Idles, flow control primitives, and begin of frame and end of 
frame symbols. 

35 Each frame buffer 166, 168 is preferably large enough to 
store hundreds of Fibre Channel (FC) or Gigabit Ethernet 
frames. For example, in a preferred embodiment each frame 
buffer 166, 168 is sufficiently large to allow 240 full sized 
FC frames (of 2148 bytes each including headers, CRC and 

^ delimiters) to be stored. As a result, the link card 140 is able 
to accept from each data channel of the client device up to 
240 full size Fiber Channel (FC) frames more than the next 
downstream device has indicated it is ready to accept. 
The link interface processor 164 also translates protocol 

45 specific frame delimiters, idle Words, and link synchroniza- 
tion characters into generic counterparts that are sent 
through the rest of the fiber optic ring network 100. As a 
result, the operation of all components of the fiber optic ring 
network other than the link interface processors operate in a 

50 manner that is protocol independent. In the case of Fibre 
Channel link cards, the link interface processors translate 4 
byte idles and link synchronization words into 2 byte generic 
versions, which are written to the inbound frame buffer 166. 
Similarly when the 2 byte generic versions of these symbols 

55 are read from the outbound frame buffer 168, they are 
converted back to the 4 byte Fibre Channel versions, with 
the reading of the outbound frame buffer paused as neces- 
sary to align data frames to the 4 byte boundaries. 

The Mux unit interface 154 includes a Mux interface 

60 frame processor 170 that controls the flow of data between 
the frame buffers 166, 168 and the Mux units 142 (Mux 1 
and Mux 2 of FIG. 3). The Mux interface frame processor 
170 also decodes 10ft symbols in the two data streams 
received from Mux 1 and Mux 2 into 8b symbols, and 

65 encodes 8b symbols received from the frame buffers 168 
into 10b symbols suitable for transmission over optical fiber 
cables. 
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The Mux interface frame processor 170 handles flow data to and from client device interface 0 from and to one of 

control as follows. When distance buffering is disabled, the the MUX's (i.e., either MUX 1 or MUX 2). Client device 

MUX interface frame processor 170 passes data frames and interface 1 data then would be routed to and from the other 

flow control primitives through the inbound and outbound MUX (142) not being used by client device interface 0. 

frame buffers 166, 168 as quickly as possible. If a link card 5 In an alternate embodiment, or an alternate configuration 

channel is in FC mode, the MUX interface frame processor of the preferred embodiment, the MUX interface frame 

170 needs to wait for the inbound frame buffer 166 to collect processor 170 is configured to route frames from both client 

enough (i.e., a predefined amount) of the FC frame before it interfaces 159 to the same MUX 142 and the frames would 

starts transmitting the frame to the MUX 142 to avoid a be specially tagged so they could be sent out to appropriate 

buffer underrun condition. This is because the MUX unit 10 client device, via the appropriate client device interface at 

interface 154 always operates at 1 .25 Gbps and the FC link * e °* er ««J of ^ other woul ? th 5 n be 

interface operates at 1 0625 Gbps. To avoid an underrun ™ d for m . casc 1 ? a ' uh £ m \. hard !"? 

j *. • • ™ , mi v • 4 j component in the original path. Routing frames from both 

condition in FC mode the MUX interface processor 170 ^ ^ ^ 1J9 * ^ MUX P ^ 

T ™ £*£ ^tJTf 411 • , Y e £ wnen ««* to*™ have be « compressed by the link interface 

inbound FIFO buffer 166 before starting to read the frame, 15 &ame processor s 164 (i.e. , in embodiments in which the link 

or until that much time has elapsed to handle the case when processors 164 mclude data compression 

the frame is less than 384 bytes in length. In the case of circuitry). 

Gigabit Ethernet, there is no need to wait before starting to Each link caid 140 ^ includes a CPU or controller 180 
read the frame out of the buffer since the clock speeds of the for controlling the operation of the link card 140, and, in 
client device and the Mux unit interface 154 are matched. 20 particular, for configuring the data paths through the link 
When a link card channel has distance buffering enabled, card 140 and for initializing the link card 140 upon power 
the MUX interface frame processor 170 executes a flow up, reset, or a change in the data channel configuration of the 
control protocol with the link interface frame processor 164 system that changes the data paths through the link card 140. 
for that channel. For instance, if the outbound frame buffer In some preferred embodiments, the link interface frame 
168 starts filling up, this condition is detected by the MUX 25 processor 164 further performs data compression and 
interface frame processor 170, which responds by sending decompression functions, compressing outbound data 
flow control signals to the MUX interface frame processor streams using a predefined data compression and decom- 
170 in the "sending link card" (connected to the sending pressing inbound data streams using a corresponding 
client device), which then stops sending frames over the decompression method. Numerous appropriate data corn- 
fiber optic network, leaving them in the sending link card's 30 pression methods are well known to those skilled in the art, 
inbound frame buffer 166. Once the outbound frame buffer and thus are not described here. By using a data compression 
168 starts clearing (i.e., the receiving client device accepts method that, on average, achieves at least 2:1 data 
enough frames to bring the outbound frame buffer's fullness compression, the bandwidth of the system can be doubled, 
below a predefined level), the MUX interface frame proces- Mux Interface Frame Processor 

sors 170 in the sending and receiving link cards will 35 ^„ „ .„ _ , 

exchange flow control messages (using flow control ^ mG ' 6 Urates a block diagram of the Mux interface 

primitives) and allow the data frames to start flowing again. *™? P ro f ssor 170. ^e Mux mterface frame processor 

„„_ ' L . . . , , 170 includes a pan of FIFO read circuits 181: a first FIFO 

When link synchrenizaUon character are received, only read bbck m _ t ided for ^ mterface Q pjpo 

one of them is ^stored in the inbound HFO frame buffer 166. ^ ft pjpQ read Wock 181 _ 2 ided for lhe 

The Mux interface ^ frame processor 170 when reading data t ^ nm read 1SV1 m . 2 is 

from ^the inbound FIFO frame ^buffer 16 , replicates the link confi d to control the readm of the correspOD ding 

synchromzation characters and/or Idles, for as many times as mbouQd mo framfi bufifcr J66 ^ ^ If ^ ^ nQm 

may be needed to fill the data stream being sent downstream. ^ nQ md nQ Qther dMa othef ^ [d{cs Qr ^ 

The Mux unit interface 154 draws data from the inbound 45 initialization primitives) in the buffer 166, then the FIFO 

frame buffer 166 and sends it to the Mux units 142 at a fixed rea d circuit will repeatedly output the most recently read 

rate for so long as the Mux units 142 are able to accept data link initialization word or Idle word. Each FIFO read circuit 

and there is data in the inbound frame buffer 166 to be sent. igi.^ igl-2 also holds off reading from its corresponding 

In the outbound direction, the Mux unit interface accepts inbound FIFO 166 if the corresponding Inband Tx circuit 

data from the Mux units 142 as long as the outbound frame 5Q ig 5 _i or ig5_ 2 ^ processing a pending request or if the 

buffer 168 has room to store at least one additional full size corresponding Rx Credit logic 190-1 or 190-2 is enabled and 

frame (e.g., 32 k bytes for Gigabit Ethernet frames), and indicates there is no credit available. In FC mode the FIFO 

stops accepting data (i.e., new frames of data) from the Mux rea d circuit 181 delays reading a packet from the inbound 

units 142 once the outbound frame buffer 168 passes that piFO until enough of the frame is present to ensure that an 

predefined fullness mark 55 un derrun will not occur (as already discussed above). 

The Mux interface frame processor 170 is coupled to each The pair of FIFO read circuits 181-1, 181-2 are coupled 
of the two multiplexers Mux 1 and Mux 2 (142) by a to a pair of multiplexors 183-1, 183-2 configured to allow 
respective serializer/deserializer (SERDES) circuit 174 such Interface 0 to be statically connected to MUX 1 or MUX 2 
as the Vitesse VSC7135. The SERDES 174 converts the and Interface 1 to be connected to the other MUX. As 
serial data stream from a multiplexer 142 into a 10 bit 60 indicated earlier, this MUX configuration can be enhanced 
parallel data stream that is transmitted to the Mux interface to allow the data streams from both client devices interfaces 
frame processor 170, and converts a 10 bit parallel data to be blended and routed through a single MUX 142 (MUX 
stream received from the Mux interface frame processor 170 1 or Mux 2), for instance during failover. 
into a serial data stream that is transmitted to one of the Mux Subsequent to each MUX, the Mux interface frame pro- 
units 142. 65 C essor 170 also includes a series of circuits blocks: an 

In the preferred embc)diment, the Mux interface frame Inband Tx circuit 185, a TX credit circuit 186 and a Sb/lOb 

processor 170 is statically configured by the user to route encoder 187. 
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Each Inband Tx circuit 185 includes registers to hold a 
pending Inband frame that the onboard CPU 180 wishes to 
send and arbitration logic to send the Inband frame when the 
FIFO Read circuit 181 signals that there is a gap between the 
inbound frames being transmitted through the Mux interface 
frame processor. Inband frames are periodically transmitted 
by the link card 140. The Inband frames are removed from 
the data stream by the Inband Rx circuit 189 in the receiving 
link card, which sends the received Inband frames to that 
link card's local CPU/controller 180 for processing. In this 
way the CPU's 180 on the two link cards at the two ends of 
a communication channel can send messages (i.e., in Inband 
frames) back and forth to check on the functionality of the 
channel and to coordinate user settings (i.e., to make sure the 
user settings, including the user setting indicating whether 
distance buffering is enabled or disabled, are the same) and 
the like. 

Each Tx Credit circuit block 186 is configured to insert 
link credit words onto the link instead of Idles when distance 
buffering is enabled and there is room in the outbound frame 
buffer 168 (FIG. 5) to store an additional packet, taking into 
account all previously sent link credit words. 

Each 8b/10b encoder 187 is configured to encode the 16 
data bits and 1-bit k-character flag read from the FIFO into 
two 10 bit characters and to send the resulting 20 bit word 
to the SERDES 174. 

The data receiving circuitry of the Mux interface frame 
processor 170 includes a cascading chain of three circuit 
blocks for each receive channel: an Rx DataPath circuit 191, 
an RX Credit circuit 190 and an Inband Rx circuit 189, The 
Rx DataPath circuit 191-1 is substantially identical to the Rx 
DataPath circuit 191 in the TDM smoother in MUX1 and 
MUX2, and will be discussed below with respect to FIG. 11. 
The Rx Credit circuit 190 strips link credit words from the 
data stream and for each such link credit word adds to the 
available storage credit, if distance buffering is enabled. The 
storage credit accumulated by the Rx Credit circuit 190 
indicates how many data frames the corresponding FIFO 
read circuit 181 can read from its inbound FIFO and send 
down the channel to the link card on the other side of the 
channel. As indicated above, the Rx Inband circuit 189 strips 
inband frames from the data stream and stores them for 
reading by the link card's local CPU 180. 

A pair of outbound multiplexors 184-1 and 184-2 are 
configured to allow MUX 1 to be statically connected to 
client device Interface 0 or Interface 1 and MUX 2 to be 
connected to the other Interface. As stated above, this MUX 
configuration can be enhanced to allow the data streams 
from both client devices interfaces to be blended and routed 
through a single MUX 142 (MUX 1 or Mux 2). 

The output of each multiplexors 184 is sent to a FIFO 
Write circuit 182, which writes received frames to the 
outbound FIFO frame buffer 168 (FIG. 5) and also writes 
link initialization words and Idles to the buffer 168 when 
they are different from the immediately preceding words in 
the data stream. 

The Mux interface frame processor 170 further includes 
Status and Control Registers 192, which are a set of registers 
that are readable and/or writeable by the link card's local 
CPU 180 in order to monitor and control the Mux interface 
frame processor. 

Link Interface Frame Processor 

In a preferred embodiment, each communication channel 
is either in Fibre Channel (FC) mode or Gigabit Ethernet 
(GE) mode. As more protocols come into common usage, 
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the number of such modes will increase. A different version 
of the link interface frame processor 164 (FIG. 5) is pro- 
vided for use with client devices transmitting and receiving 
data in each mode. A FC link interface frame processor 
5 depicted in FIG. 7 is provided for FC mode, and a GE link 
interface frame processor depicted in FIG. 8 is provided for 
GE mode. 

FC Link Interface Frame Processor. 

Now referring to FIG. 7, the FC link Interface Frame 
10 Processor 164-1 includes an FC Rx DataPath circuit 193-1 
that is substantially similar to the RX DataPath circuit 
(described below with respect to FIG. 11) used in the Mux 
TDM-Smoother of the Mux interface frame processor. 
However, the RX DataPath circuit 193-1 has additional logic 
15 at its front end to convert 4 character FC idle and link 
initialization words to predefined 2 character generic coun- 
terparts. These counterpart symbols are used only internal to 
the fiber optic ring network (i.e., within the link cards 140, 
MUX's 142 and the fiber optic cables). 

The FC link interface frame processor 164-1 further 
includes: 

An Rx Credit circuit 194-1 that strips link credit words 
(RRDY) from the data stream and for each such link 

25 credit word adds to the available storage credit, if 
distance buffering is enabled. 
A FIFO Write circuit 195-1 that writes received frames to 
the inbound FIFO frame buffer 166 (FIG. 5) and writes 
link initialization words and idles to the FIFO when 

30 they are different from the immediately preceding 
words in the data stream. 
A FIFO Read circuit 200-1 that controls the reading of 
data frames from the outbound FEO frame buffer 168 
(FIG. 5). If there is nothing in the outbound FIFO frame 

35 buffer, then the FIFO Read circuit 200-1 will repeatedly 
output the most recently read link initialization or idle 
word. If distance buffering is enabled, the FIFO Read 
circuit furthermore holds off reading data frames from 
the FIFO if no data frame credits are available. 

40 A ring to FC conversion circuit 199-1 that converts 
2-character generic idle and link initialization words 
into their standard FC 4-character counterparts. 
A Tx Credit circuit 198-1 that inserts link credit words 

45 onto the link instead of Idles when distance buffering is 
enabled and there is room in the inbound FIFO frame 
buffer 166 to store an additional data frame, taking into 
account previously send link credit words (see later 
section on distance buffering for details). 

50 An Sb/lOb encoder 197-1 that is configured to encode the 
16 data bits and 1-bit k-character flag read from the 
outbound FIFO frame buffer 168 into two 10 bit 
characters and to send the resulting 20 bit word to the 
SERDES 162 (FIG. 5). 

55 An FC Statistics circuit 196-1 that includes logic to 
maintain statistics on one Fibre Channel Link. Several 
modes of operation are supported to provide users 
detailed protocol specific information, for instance 
packet counts, error counts, counts of various specific 

go types of K characters, and so on. 

Status and Control Registers 201-1, which are a set of 
registers that are readable and/or writeable by the link 
card's local CPU 180 in order to monitor and control 
the link interface frame processor 164- 1. 

65 GE Link Interface Frame Processor. 

Referring to FIG. 8, the GE link Interface Frame Pro- 
cessor 164-2 is used on link cards that couples the client 
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devices using Gigabit Ethernet to the fiber optic network. 
The GE/IFP 164-2 includes an FC GE DataPath circuit that 
is substantially similar to the RX DataPath circuit (described 
below with respect to FIG. 11) used in the Mux TDM- 
Smoother of the Mux interface frame processor. However, 5 
the RX Datapath circuit 193-1 has additional logic at its 
front end to convert GE idle words into predefined 2 generic 
counterparts. In addition the delimiters framing Gigabit 
Ethernet frames are modified to a generic format These 
counterpart symbols are used only internal to the fiber optic 10 
ring network (i.e., within the link cards 140, MUX's 142 and 
the fiber optic cables). 

The GE link interface frame processor 164-2 further 
includes: 

RxPause logic circuit 194-2 strips Pause frames from the 15 
data stream and starts a pause timer, internal to the link 
card, if distance buffering is enabled. 

A FIFO Write circuit 195-1 that writes received frames to 
the inbound FIFO frame buffer 166 (FIG. 5) and writes 
link initialization words and idles to the FIFO 166 20 
when they are different from the immediately preceding 
words in the data stream. 

A FIFO Read circuit 200-1 that controls the reading of 
data frames from the outbound FIFO frame buffer 168 
(FIG. 5). If there is nothing in the outbound FIFO frame 
buffer, then the FIFO Read circuit 200-1 will repeatedly 
output the most recently read link initialization or idle 
word. If Rx Pause logic (which is used in conjunction 
with distance buffering) is enabled, the FIFO Read 30 
circuit furthermore holds off reading data frames from 
the FIFO if the Rx Pause logic circuit 194-2 indicates 
that transmission should be paused. 

A ring to GE conversion circuit 199-2 that converts 
2-character generic idle into Gigabit Ethernet idle 35 
words, and generic frame delimiters back into Gigabit 
Ethernet frame delimiters. 

A Tx Pause circuit 198-2 generates and inserts into the 
outbound data stream a Pause frame when distance 
buffering is enabled and the inbound FIFO frame buffer 40 
166 is at least half full. The FIFO fullness threshold 
level for generating a Pause frame may differ in other 
embodiments: See the discussion on distance buffering, 
below, for details. 

An Hb/lOb encoder 197-2 that is configured to encode the 45 
16 data bits and 1-bit k-character flag read from the 
outbound FIFO frame buffer 168 (FIG. 5) into two 10 
bit characters and to send the resulting 20 bit word to 
the SERDES 162 (FIG. 5). 

A GE Statistics circuit 196-2 that includes logic to main- 
tain statistics on one Gigabit Ethernet Channel Link. 
Several modes of operation are supported to provide 
users detailed protocol specific information, for 
instance packet counts, error counts, counts of various 
specific types of K characters, and so on. 

Status and Control Registers 201-2, which are a set of 
registers that are readable and/or writeable by the link 
card's local CPU 180 in order to monitor and control 
the link interface frame processor 164-2. 6Q 

Multiplexer Units 

Referring to FIG. 9, each Mux unit 142 includes a pair of 
wavelength division multiplexer and demultiplexer circuits 
(WDM) 202-1, 202-2, each of which is coupled at one end 65 
to a respective optical fiber cable segment, for example 
OL12-1 and OL13-1. Each of the WDM's 202-1, 202-2 
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includes an optical signal receiver for receiving and 
demodulating signals at a first optical wavelength and an 
optical signal transmitter for transmitting signals at a second 
optical wavelength. More specifically, in a preferred 
embodiment the first WDM 202-1 transmits at the same 
optical wavelength XI that the second WDM 202-2 receives, 
and receives at the same optical wavelength XZ that the 
second WDM 202-2 transmits. In an alternative 
embodiment, each Mux unit 142 is configured to handle four 
wavelengths over eight channels. The four wavelength — 
eight channel card has twice as many SERDES 208, TDM/ 
Smoother 206 and SERDES 204 circuits compared with two 
wavelength — four channel Mux depicted in FIG. 9. The four 
additional SERDES 208 are connected to the crosspoint 
switch 210. Two SERDES 204 are then connected to each of 
its Fiberl/WDM and Fiber2/WDM circuits. Fiberl/WDM 
then transmits at wavelengths XI and X3, and receives at 
wavelengths X2 and X4. Fiber2/WDM transmits at wave- 
lengths and X4, and receives at wavelengths XI and X3. 

Each of the data signals received and transmitted by the 
WDM's 202-1, 202-2, both on the optical side and on the 
electrical (internal) side are 2.5 Gbps serial data signals in 
the preferred embodiment. In other embodiments, other data 
transmission rates may be used, such as 5.0 Gbps or 10 
Gbps. 

Each WDM 202 is coupled to a respective time division 
multiplexer and smoothing circuit (TDM smoother) 206 by 
a respective serializer/deserializer (SERDES) circuit 204, 
such as the Vitesse VSC7146 (for data streams running at 2.5 
Gbps). Each SERDES 204 converts the 2.5 Gbps serial data 
stream from its respective WDM 202 into a 20 bit parallel 
data stream that is transmitted to the TDM smoother 206 to 
which it is coupled, and converts a 20 bit parallel data stream 
received from the TDM smoother 206 into a serial data 
stream that is transmitted to the WDM 202. 

The TDM smoother 206 performs a number of functions, 
including retiming of signals between clock domains, the 
multiplexing of data streams from two 1 .25 Gbps channels 
into a single 2.5 Gbps data stream, and the demultiplexing 
of data streams from a 2.5 Gbps data stream into two 1.25 
Gbps channels. The TDM smoother 206 is described in more 
detail below, with reference to FIGS. 10 and 11. 

Still referring to FIG. 9, the TDM smoother 206 internally 
uses 20b parallel data streams. On its Channel AB interfaces 
(which are coupled to the WDM's 202) it outputs and 
receives 20b parallel data streams. On its Channel A and 
Channel B interfaces, which are coupled to a crosspoint 
switch 210, the TDM outputs 10b parallel data streams. A 
pair of SERDES circuits 208, such as the Vitesse VSC7135, 
are coupled to the switch side of each TDM to convert the 
106, 125 MHZ, data streams going to and from the switch 
side TDM interface into 1.25 GHz (i.e., 1.25 Gbps) serial 
data signals that are transmitted to and from the crosspoint 
switch 210. In a preferred embodiment, the crosspoint 
switch 210 is a 16x16 crosspoint switch, such as the Triquint 
TQ8017 1.25 Gbps 16x16 digital crosspoint switch. There is 
one crosspoint switch 210 in each Mux unit 142. 

The 2.5 Gbps signal received by the Mux unit 142 from 
each optical fiber cable includes two 1.25 Gbps data signals, 
which in turn may be sub -divided into two or more logical 
signals. Each 1.25 Gbps data signal is considered to be a 
separate logical channel, and each such channel may be 
either an FC channel or a GE channel. The two data channels 
on a single optical fiber cable may be an two FC channels, 
two GE channels, or one FC channel and on GE channel. 
Since FC and GE data streams are both converted into a 
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generic data stream that is protocol independent, the two character because it has a predefined "comma" bit pattern 

data channels within each 2.5 Gbps signal can be any ("0011111") used by deserialization circuits to align 

combination of underlying data streams. received bit streams to word boundaries. 

If an entire 1.25 Gbps data signal is to be forwarded by a Another special character, called the K28.1 character, also 

network node to a next network node, that data stream is 5 contains the comma bit pattern. When combining two or 

routed from a first optical fiber cable up through a WDM 202 more data streams for transmission over an optical fiber 

and TDM smoother 206 to a crosspoint switch port, through cable > me unk multiplexer marks a first one of the data 

the crosspoint switch to a second switch port, down through streams by replacing all its K28.5 characters with the K28.1 

the other TDM smoother 206 and WDM 202 of that mul- character, which enables the receiving device to separate out 

tiplexer unit, and back out onto a second optical fiber cable 10 and properly identify the different logical data streams with 

for transmission to the next network node. ^ e received physical data stream. 

Each Multiplexer unit 142 includes a CPU or controller ^ ^ mbound data paths each begins with a latch 230 

212 for configuring the switch 210 and keeping track of the ^t stores every second 10-bit ^ character , which is then 

status of the TDM smoothers 206 and WDM's 202. combined with me immediately following 10-bit characters 

15 to form a stream of 20-bit characters. The 20-bit characters 

TDM Smoother Circuit are transmitted through ao Rx Datapath circuit 232-1 or 

232-2 to a time division multiplexer (TDM) 234. The Rx 

FIG 10 shows j more detailed Diagram of the TDM DauPam drcuit 232 is described in more detail below with 

smoother circuit 206. The left side of the diagram represents reference to FIG 12 

the switch side interface between the TDM smoother 206 2Q ^ TOM 234 combines the Channel A and Channel B 

and the SERDES circuits 208 (FIG. 9), while the right side daU strcams using strict ^^3^ Th at ^ it alternates 

of the diagram represents the WDM side interface between between transmitting a 20-bit character from Channel Aand 

the TDM and the SERDES circuits 204 (FIG. 9). The a 2fJ _ bit charactcr £ om channel q For mstancc , on evcn 

mbound data path through the TDM smoother 206 converts numDered clock cycles Chanrjel A data is ^ hcUd by the 

the Channel A and Channel B Rx data streams into a ^ ^DM 234 for transmission, and on odd numbered clock 

combined Channel AB Tx data stream, while the outbound les cb aniie i B data is selected by the TDM 234 for 

data path through the TDM smoother 206 converts the transmission 

Channel AB Rx data stream received from a WDM circuit [n order for a device at ^ receiving end to know which 
into a pair of Channel A and Channel B Rx data streams. cycle fa ^ eyen cyde ^ which fa the odd ^ the TOM 
In order to explain the operation of the TDM smoother 30 234 replaces all the K28.5 characters in Channel A with 
206, some background information is required. Regardless K28.1 characters. In an alternate embodiment, in which 
of the protocol of the signals received by the link cards 140 multiple logical channels (Channel A through Channel n) are 
of the link multiplexer 106 (FIG. 3), the incoming data is combined by the TDM 234 into a single signal for trans- 
converted by the link cards into a 1.25 Gbps stream that has mission over an optical fiber cable, the TDM 234 marks a 
frames of "encapsulated" data (i.e., surrounded by special 35 first one of the logical channels by replacing its K28.5 
start and end frame characters). Each frame of data that is characters with K28.1 characters, which enables the receiv- 
transmitted through the link multiplexer begins and ends mg device to identify all the logical channels within the 
with a special 20-bit encapsulation character. In addition, the received signal. The K28.5 characters in all the other logical 
link multiplexer transmits 20-bit flow control characters channels are left unchanged. 

between fink cards to stop and start the flow of frames 40 Since every link multiplexer 106 in the system uses 
between the FIFO frame buffers in the link cards. identical multiplexer units, all the data streams transmitted 
Data characters, which are 10-bits each, are transmitted over the optical fiber cable segments use the same marking 
through the link multiplexer in pairs in 20-bit chunks. Thus scheme for distinguishing between a first subchannel within 
the basic unit of transmission in the link multiplexer, both each data stream and the other subchannel(s) in the same 
for data and control characters, is 20-bits long. 45 data stream. Since the 20-bit data streams are combined 
The set of predefined 20-bit control characters used in the "blindly,'* there can be "false commas" straddling the bound- 
link multiplexer of the preferred embodiment includes, but ary between the two disparate 20-bit characters. As a result, 
is not limited to the following: the SERDES circuits in the link cards and multiplexer units 
frame encapsulation characters (start and end of frame); arc 0111 ^eir "comma detect" mode disabled, except 
flow control characters (ready to receive an additional 50 durin S J** initialization, so as to prevent the SERDES 



frame); 



circuits from accidentally realigning the received bit streams 

^J^^k^^^^T' ^in^ CSlaWiSh PatbS S e mbound a data path within the TDM smoother 206, 
through the link multiplexer 106; a time division demultiplexer (TDDM) 240 receives a 20-bit 
"Internal Violation" (VIO) characters, used to indicate the 55 125 MHZ signaJ ^ received signal contains two logical 
presence of invalid words in the middle of a frame; subchannels, in the form of alternating 20-bit characters. 
"Internal Not Operational Sequence" (NOS) characters, T^e TDDM 240 inspects the received signal to (A) find the 
used to indicate a lack of synchronization; and 20-bit word boundaries in the signal and (B) determine 
Idle characters, which are transmitted whenever there are which logical subchannel has K28.1 characters and is there- 
no data or other characters to be transmitted over any 60 fore Channel A- The TDDM 240 transmits the 20-bit char- 
of the communication channels in the link multiplexer. acters from first subchannel through a first Rx Datapath 
Each of the special 20-bit characters used in the link circuit 232-3 to a first output buffer 244- 1, and transmits the 
multiplexer consists of a predefined K28.5 10-bit character, 20-bit characters for the second subchannel through a sec- 
followed by a 10-bit character specific code. The K28.5 ond Rx DataPath circuit 232-4 to a second output buffer 
character is the "special" character most commonly used in 65 244-2. The output buffers 244 each convert a received 
Fibre channel and Gigabit Ethernet to denote control stream of 20-bit 62.5 MHZ data signals into a 125 MHZ 
characters, as opposed to data characters. It is an ideal stream of 10-bit characters. 
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Each of the Rx DataPath circuits 232 receives a stream of 
20 bit symbols, but outputs a data stream of 18 bit symbols, 
each of which includes 16 bits of data, one flag to indicate 
if the first 8 bits need to be encoded as a "K character/' and 
a valid bit to indicate if the data word is valid or invalid. 5 
These 18 bits are then encoded by a respective one of the 
SbflQb encoders 242. 

Hie Channel A data path circuit within the RX Datapath 
circuit also has an Inband Tx circuit 246 for inserting special 
inband control frames into the data stream during the idle ]Q 
times. Control information can be distributed to the control- 
lers in the link cards and multiplexer units of a fiber optic 
network by a single computer system or a single node 
controller on the fiber optic network. The controllers within 
the network system communicate with each by having the 
control CPU's 212 of the MUX units 142 (and the control 15 
CPU's 180 of the link cards 140) send these inband frames. 
The control CPU 212 writes the frame to a 64 byte register 
inside the Inband Tx circuit 246. The control CPU then 
writes a flag to tell the hardware that the frame is ready to 
go. When the Rx DataPath 232-1 indicates there is enough 20 
idle space to insert a control frame (as detailed in more 
below), the Inband Tx circuit 246 inserts the control frame 
with a special start of frame delimiter onto the Datapath 
instead of Idles. 

When the inband control frame arrives at another MUX 25 
unit, after transmission over an optical fiber cable, the 
Inband Rx circuit 248 detects the special start of frame 
delimiter and stores the 64 byte frame data into the next one 
of eight Rx buffers (included in Status and Control Registers 
254), The Inband Rx circuit 248 propagates Idles instead of 3Q 
the inband control frame data to the subsequent 86/106 
encoder 242-3. Once the entire Inband frame is received, the 
Inband Rx circuit marks the Rx buffer into which the frame 
was written as being in use and signals the control CPU 250 
that an Inband control frame is available. Once the control 
CPU 212 has read the frame, it marks the Rx buffer as 35 
available. If a special inband frame is received and the next 
Rx buffer is not available, the inband frame data is discarded 
by the Inband Rx circuit 248. 

The TDM smoother 206 also includes a set of status and 
control registers 254 that are read by the Mux unit's CPU 40 
212 via a CPU interface 252. 

In an alternate embodiment, the TDM 234 does not 
change the K character symbols of one of the data streams 
so as to mark the A and B channels. Instead, the link cards 
of the system insert immediately before each frame a special 45 
Start of Packet (SOP) K character, replacing the Idle that 
immediately precedes the frame with SOP symbol. This task 
is performed by logic and the front end of the Rx DataPath 
circuit, and in particular in the FC Rx DataPath circuit 15*3-1 
and GE Rx DataPath circuit 193-2. The TDM 234, upon 
receiving an SOP symbol from the Channel A data path 
converts that symbol into a SOP1 symbol, thereby marking 
the data in Channel A as the first data channeL The TDDM 
240 inspects the received signal to (A) find the 20-bit word 
boundaries in the signal and (B) determine which logical 
subchannel has SOP1 characters and is therefore Channel A 5S 
The TDDM 240 transmits the 20-bit characters from first 
subchannel through a first Rx DataPath circuit 232-3 to a 
first output buffer 244-1, and transmits the 20-bit characters 
for the second subchannel through a second Rx DataPath 
circuit 232-4 to a second output buffer 244-2. Furthermore, 60 
the TDDM 240 converts the SOP and SOP1 symbols back 
into Idle symbols, since these special symbols are only for 
internal use within the fiber optic network. 

Rx DataPath Circuit 

Referring to FIG. 11, the Rx DataPath circuit 232, (of 
which there are four instances in each TDM smoother 206, 
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FIG. 10, and two instances in each MUX interface frame 
processor 170, FIG. 6), receives a 20 bit signal converts it to 
16 bit data, a K character flag and an invalid word flag. 

The Rx DataPath circuit 232 replaces any invalid words 
that are in a frame with a special violation word (FVIO), 
eliminates any invalid words that are outside of a frame, and 
retimes the data stream onto the local clock of the link card 
or MUX unit. It also maintains a count of invalid words 
received so that failing links can be easily isolated. 

Each received 20 bit word is initially decoded into 16 bit 
data and flags by a 10b to Sb decoder circuit 274. The 
decoder circuit 274 produces a K-character flag plus a valid 
flag that indicates whether the 20 bit word was made up of 
valid 10 bit codes. The 16 bit data and flags are sent to a 
word decoder and loss of synch state machine 276. 

The word decoder 276 keeps track of whether the 
received data is inside a frame or outside of frame by 
recognizing the start of frame and end of frame delimiters. 
If the received word is valid, the 16 bit data and K character 
flag are passed as is to a drop circuit 278. If the received 
word is invalid and the data is in the middle of frame, the 
word is replaced with the special FVIO word. Downstream 
logic will recognize that this is not the original data, but it 
will not count it as an invalid word to facilitate error 
isolation, because it is not known where along data path the 
error occurred except that it occurred at a node prior to the 
receiving node. If the received word is invalid and the data 
is not in a frame, then a Force Drop flag is asserted to the 
drop circuit 278 so that the invalid word will be dropped 
completely from the data stream. 

If the state machine 276 detects four invalid words within 
any ten consecutive words, the state machine 276 assumes 
that the received data stream has lost synchronization. In this 
case it will propagate an FNOS word to the drop circuit 278, 
marked with a K-character flag and a Insert/Drop OK flag. 
After this, the state machine inspects the incoming data 
stream and replaces each word in the data stream with an 
FNOS word until it receives three consecutive valid words 
that are either Link Initialization words or Idles, at which 
point the state machine 276 assumes that synchronization of 
the received data has been re-established and resumes propa- 
gating words from the data stream to the drop circuit 278. 

In addition, the word decoder and los of synch state 
machine 276 determines if the received word is an Idle or 
one of a predefined set of four link initialization words. 
When any of these five symbols is detected, the state 
machine 276 sets a corresponding one 5 idle/init decode 
flags and also sets the Insert/Drop OK flag. 

The 16 bit data, K character flag, 5 idle/init decode flags 
and the Insert/Drop Ok flag are passed through a 23 bit wide 
FIFO 280. In a preferred embodiment, the FIFO 280 stores 
up to 128 words, each 23 bits wide. 

The drop circuit 278, 128x23b FIFO 280 and an insert 
circuit 282 form a smoother or data retiming circuit. The 
drop circuit 278 and the write side of the FIFO 280 operate 
on the Rx Clock (recovered by the external SERDES circuit 
from the serial receive data). 

The insert circuit 282 and the read side of the FIFO 280 
operate on a System clock that comes from a local oscillator. 
Nominally, both of these clocks operate at the same 
frequency, but in practice they will be slightly different, and 
thus the need to retime the data stream. 

The drop circuit 278 normally writes to the FIFO 280 
every clock cycle. However if the Force Drop flag is on (i.e., 
set), or if the FIFO 280 is more than half full and the 
Insert/Drop Ok flag is on, the FIFO write enable will be 
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suppressed and the current word from the decoder 276 will In the alternate embodiment in which SOP and SOP1 

be discarded (i.e., it will not be written into the FIFO 280). symbols are used to mark the Channel A and Channel B data 

The insert circuit 282 normally reads from the FIFO 280 streams, the Rx DataPath circuits 193-1 and 193-2 in the link 

every cycle. However if the FIFO 280 is less than one cards have a slightly modified word decoder and state 

quarter full and the last word read from the FIFO 280 had the S machine 276. In this embodiment, the word decoder 276 

Insert/Drop OK flag set, the FIFO read is suppressed and last includes a FIFO, which can have a length of one or two 

read word is replicated onto the output. words, that enables the word decoder to perform a look 

Thus if the Rx clock is slightly faster than the System for start of &ame ( S0F ) symbols preceded by an Idle. 

clock, the FIFO 280 will occasionally go past half full since Whenever this combination of two symbols is detected by 

the rate of reads from the FIFO is slightly slower than the 10 & c v™* 6 decoder 276, the word decoder replaces the Idle 

rate of writes to the FIFO. The drop circuit 278 will then immediately preceding the SOF with an SOP symbol. As 

occasionally drop words to keep the FIFO 280 less than half explained above, the SOP symbol in Channel A is replaced 

full. If the System clock is slightly faster than the Rx Clock, b y S0P1 symbol by the TDM circuit 234 in the TDM 

then the FIFO will occasionally go below one quarter full smoother circuit of a Mux unit. 

since the rate of reads from the FIFO is slightly faster than 35 ^ - - 4 . cn . VT ^ . 

«. 4 r tl - . • * 4 t ty% .„ 4 Static Configuration of Ring Network 

the rate of writes. The insert circuit 282 will then insert a & & 

word into the data stream to keep the FIFO above one During normal operation, the data stream paths through 

quarter full. the ring network are statically configured. That is, the signal 

In addition, the insert circuit 282 has some special fea- paths are not constantly being determined on the fly. Rather, 

tures to support the transmission of Inband data. When the 20 it is assumed that the customers using ring network lease 

Inband Tx circuit 246 (e.g., of the TDM smoother) has an bandwidth on the network on an ongoing basis. Signal paths 

inband frame to transmit, it will assert an Inband Tx request through the network are, in general, changed only when (A) 

signal. If the FIFO 280 is less than half full, and the current there is a change in the set of leased channels on the 

word has the Insert/Drop OK flag on, the insert circuit 282 network, or (B) there is a link failure. 

stops reading from the FIFO 280 and sends an "Inband Tx 25 

go" signal to the Inband Tx circuit that is immediately Distance Buffering 

downstream from the RX DataPath circuit 232. The insert ^ many ^plications, the host devices (also called clients 

circuit continues to replicate the current word on its output or client devices) communicating via the ring network are 

for several clock cycles, until the entire pending Inband 3(} many kilometers apart. For instance when two devices are 

frame has been inserted into the data stream by the Inband fifty kilometers apart, with a round trip communication path 

Tx circuit, 0 f ^qq kilometers, the round trip communication time is at 

While the Inband Tx go signal is asserted, the downstream least 500 microseconds, excluding the time it takes for the 

inband Tx circuit will rep lace the data from the Rx Datapath receiving device to receive and respond to an incoming 

circuit with the Inband Tx frame. Once the inband frame 35 signal. If the input buffers of the receiving device are small 

transmission is complete, the Inband Tx circuit de-asserts (e.g., 8k bytes), the effective bandwidth of a 1.0625 Gbps 

the Inband Tx request signal, and the insert circuit 282 channel may be much smaller than the full bandwidth. For 

resumes normal operation. After an Inband frame has been instance, consider a system in which a client device requests 

sent, the FIFO 280 will often be more than half full, and files from a disk farm at a node that is fifty kilometers away,* 

therefore the drop circuit 278 will drop as many words as 4Q and the requesting client's input buffers only hold 8k bytes 

possible to bring the FIFO back under half full. (i.e., about four Fibre Channel frames). When the client 

Since the Inband frames are 64 bytes in length and half of sends its initial data request, it also sends four storage credits 

a FIFO holds 128 bytes (64 two byte words), the FIFO 280 to the disk farm node. It does not send more than four credits 

will not be overrun while inband transmission is in progress, because that would cause the disk farm to send more data 

since the inband transmission will not start until the FIFO 45 than the client can reliably buffer and process, which would 

280 is less than half full. result in costly resend requests and large delays. 

Another function of the insert circuit 282 is to set a valid The disk farm, using prior art methodology, responds by 

output flag for use by the MUX interface frame processor sending only four FC frames of data and waits until it 

instances of the Rx DataPath circuit 232. The insert circuit receives more storage credits from the requesting client 

282 sets the valid output flag whenever (A) the word read 50 before sending any more. However, it takes at least 500 

from the FIFO does not have its Insert/Drop OK flag on, or microseconds for the client to receive the first data and 

(B) the word read from the FIFO is not an Idle or link return another three credits. Thus, at best, the client is able 

initialization word that is the same as the previous word as to receive 8k bytes per 500 microsecond period, or a total 

determined by the 5 idle/init flags passed through the FIFO. data rate of about 16 Megabytes per second, as opposed to 

The MUX interface frame processor uses the valid output 55 the 100 Megabytes per second bandwidth of the channel, 

flag to determine what words need to be written to the Tnus, in this example, about 84% of the available bandwidth 

Outbound frame buffer 168 (FIG. 5). The Rx DataPath valid is wasted due to the long round trip time required for sending 

output flag is not used by the TDM smoother 206. storage credits to the sending device. This performance can 

It is noted that the use of a deep FIFO in the Rx Datapath be improved by increasing the size of the requesting client's 

circuit makes it simple to use half full and quarter full flags 60 m P u * buffers, as well as by sending a new storage credit as 

to control the insert and drop circuits. Making the FIFO 280 soon as each frame is received. 

sufficient wide to propagate the Insert/Drop OK and the five In the present invention, bandwidth usage is improved by 

Idle/Init flags avoids having to redecode the words on the providing frame buffers 166, 168 (FIG. 5) in the link cards 

output of the FIFO. Alternate embodiments may have dif- and then splitting the flow control into three separate 

ferent technology constraints and choose to implement the 65 domains. The domains are (1) client device to fink card, (2) 

same functionality using a shallower and narrower FIFO link card to link card across the fiber optic network, and (3) 

with additional decode logic. link card to client device. As long as the buffering in the 
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client devices is sufficient to handle the round trip link time 
from the client device to the link card full bandwidth can be 
maintained, in part because of the large frame buffers 166, 
168 provided in the link cards and in part due to the use of 
the link cards as the senders and receivers of storage credits. 5 

Based on the number of frames in the Inbound Frame 
Bufferl66, the Link interface frame processor will issue flow 
control primitives to the attached client device to maintain 
maximum bandwidth while ensuring the Inbound Frame 
buffer does not overflow. Based on the flow control primi- 10 
tives issued by the attached client device the Link interface 
frame processor will control the reading of the Outbound 
frame buffer 168. 



Details of Distance Buffering for Fibre Channel 
link Cards 



35 



After a Fibre Channel Link is initialized, Fibre Channel 
Devices perform a login procedure that includes exchanging 
how many buffer to buffer credits they have. The number of 
buffer to buffer credits advertised by a first client at one end 
of a Fibre Channel link is the number of frames a second 
client, attached to the first client by the link, can send to the 
first client before it needs to wait for additional credit. 
Additional credits are transferred by sending a special word, 
called RRDY. One RRDY word transfers one credit, which 
enables the receiving device to transmit on Fibre Channel 
frame. 

The fiber optic network of the present invention allows 
the login procedure between the two attached client devices 30 
to complete without modifying the information exchanged. 
The link cards of the system do, however, determine the 
number of buffer to buffer credits supported by the devices 
at each end of the link by examining the login frames as they 
pass through. 35 

Referring to FIG. 7, the inbound frame buffer 166 and 
outbound frame buffer 168 can each hold a maximum of 240 
maximum sized Fibre channel frames. Whenever a frame is 
written to the inbound frame buffer 166, the RX__credit 
circuit 194-1 increments a "pending RRDY" counter inter- 40 
nal to the link card. Whenever the outbound link to the client 
device is idle, the inbound frame buffer is less than half full, 
and the pending RRDY counter is greater than zero, the 
TX^Credit circuit 198-1 inserts a Fibre channel RRDY 
word into the Tx data stream and decrements the pending 45 
RRDY counter. If the inbound frame buffer is more than half 
full, the RRDY's are held pending until the inbound frame 
buffer drops below half full. Thus there is room in the 
inbound frame buffer for up to 120 frames to arrive without - 
any additional RRDY's being sent. The attached FC client 50 
device is actually operating under the assumption that it can 
only send as many frames as was specified by the client 
device on the remote end of the fiber optic network. As long 
as the remote device's buffer to buffer credit is less than or 
equal to 120 frames, the flow control between the local client 55 
device and the inbound frame buffer will operate properly. If 
the advertised buffer credit (of the remote device) is greater 
than 120, then distance buffering can be disabled, in which 
case all frames and RRDY's will be passed through the 
system end to end with no internal buffering of storage gg 
credits. As of the year 2000, most FC client devices have 
buffer to buffer credits in the range of two to sixteen frames. 
Very few FC client devices have internal buffering for even 
as many as sixty-four FC frames. 

The flow control of frames from the Outbound Frame 65 
Buffer 168 to the client device operates as follows. The link 
card must obey the buffer to buffer credit advertised by the 



attached device during login. The TX^_credit circuit 198-1 
initializes an available credit counter to the advertised 
number as it examines the login frames. Subsequently 
whenever it sends a frame it decrements the available credit 
counter by one. Whenever the RX_credit circuit 194-1 
receives an RRDY it increments the available credit counter 
by one. As long as the available credit counter is greater than 
zero, frames are read from the Outbound frame buffer and 
sent to the client device. If the available credit counter is 
zero, then frames are held pending in the Outbound frame 
buffer until an RRDY arrives. 

Details of Distance Buffering for Gigabit Ethernet 
Link Cards 

Referring to FIG. 8, whenever the Inbound Frame Buffer 
is more than half full the Tx_Pause circuit 198-2 sends an 
Ethernet Pause frame to the attached device with the pause_ 
time field set to the maximum value. This should cause the 
attached device to stop sending Ethernet frames. Once the 
Inbound Frame Buffer is less than half full, the Tx _Pause 
circuit 198-2 sends an Ethernet Pause frame with a zero in 
the pause_time field to allow the attached device to resume 
sending frames. 

If the Rx_Pause circuit 194-2 receives a Pause frame and 
distance buffering is enabled, the pause time counter is 
loaded from the pause_time field. The pause time counter is 
decremented by 1 each 512 bit times (which is the Ethernet 
standard). If the pause time counter is greater than zero, then 
frames are held pending in the Outbound Frame Buffer by 
the FIFO read circuit 200-2. 

Details of Distance Buffering Between Connected 
Link Cards 

The previous two sections have described the details of 
the flow control operation between the link cards and the 
attached Fibre Channel or gigabit Ethernet devices. Addi- 
tionally there is a flow control mechanism between the 
MUX interface frame processors on the two link cards that 
are connected across the fiber optic network. The same 
mechanism is used independent of whether the link is being 
used for gigabit Ethernet or Fibre Channel. 

Referring to FIG. 6, the link card to link card flow control 
operates in a fashion very similar to the standard Fibre 
Channel flow control mechanism. At initialization time the 
MUX interface frame processors 170 assume the attached 
link card has 120 buffer credits available. The available 
buffer credit is decremented by 1 (by the Tx credit circuit 
186) each time a frame is sent from an inbound frame buffer. 
The available buffer credit is incremented by 1 (by the Rx 
credit circuit 190) each time an intra-network buffer credit 
word (called FRRDY for "Finisar FRRDY") is received. If 
the available buffer credit is zero, frames are held pending in 
the inbound FIFO frame buffer. 

As long as the outbound FIFO frame buffer is less than 
half full an FRRDY intra-network buffer credit is sent back 
across the network (by the Tx credit circuit 186) each time 
a frame is written to the outbound FIFO frame buffer. If the 
outbound frame buffer is more than half full, the FRRDY 
intra-network buffer credits are held pending (and are sent 
once the outbound FIFO frame buffer becomes less than half 
full). 

Channel Blending 

Most communication channels are cot used anywhere 
close to 100% of capacity. In fact, most of the time channel 
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usage is well below 50%, although peak usage may reach two lines, with each line representing a distinct logical 

100%. Facilities exist in the fiber optic network of the channel. Link cards are represented by smaller boxes next to 

present invention to allow multiple external gigabit per one of the Mux boxes. A label such as "2LC: 1A2/1B2" 

second channels to be <c blended" into a single gigabit per indicates the number of link cards, and also indicates which 

second channel, which is then transmitted over the relatively 5 Mux unit ports are connected to the link cards. 

expensive long haul fibers. This allows the long haul fibers FIG. 13 shows a typical fiber optic network in accordance 

to be utilized closer to their full capacity. The present with the present invention. As shown, in many networks 

invention provides two ways to blend channels together. ^6 lhe present invention there will be a "Head End" node 

n r . . r, T « - c . i_ i .1 j- . that has individual link cards that are connected by the 

.. ? , ' 1 , ,T ^ f^L io netWOTk to link cards in several different customer nodes, 

allows the ^ochannekonasinglelink .card tobe 'blended 10 ^ ^ ^ be a „ pomt rf 

into a single channel by the Mux unit interface 154 Thus a Presence .. node for a xMce provider ^ me "Customer" 

dual channel hnk card will use only a smgle 1 25 Gbps nQdes m ^ bandwidth clients b the p oint of 

channel on one MUX unit, instead of one channel on each , n ^ , Customers 3 and 5 have two 

of two MUX unit as was described previously. Tbe two « c i eal « channels ^ lhe way to the Head End node. The other 

channelsontiKhiikcardarethencon " customers have one clear channel each, along with another 

on another link _ card. While one channel is transmitting a cleflr chumel to cust£)mer node I( should ^ under . 

frame to the MUX unit tnterface 154^any frames from toe slQod mal , he configuration shown in FIG. 13 is 

other channel are held in its Inbound FIFO fr 1 ™' ^ U ^ 6 J merely one example of many, many possible such configu- 

The large size of the frame buffers 166 on the link card give rations 

the system the ability to absorb relatively long bursts of 20 r *, . A e c .. , . 

* -sen a. • • a. u i u i * *u In order to provide a variety of failover redundancy 

frames, up to 360 frames, arriving from both channels at the , . 4 . r c , c » j j ■* u / -* L t-m_ 

«- ■ *l - 4 i j *t. j ti/u modes, it is preferred for a standard switch (either Fibre 

same time without having to slow down the senders. When- ™ , A- p.l 4 \ * . * « j • j- * i 

<i- £. ■ . »■ ,i ^ . i . 4 . , Channel or Gigabit Ethernet) to be installed immediately 

ever me frame bursts from the channels are shorter than that, , , . , A , , , 4 ' , , ~ , / 

*i_ ji 4- i . , , , t lL . , j behind the link cards at each customer node. The switch is 

the idle times between bursts are used to empty the inbound i_i j i ^ • ^ • r -i j c 

c_ . rr ^ -iw l i • t. 4 1. ■ i< enabled only during certain failover modes of operation, as 

frame buffers over the single Mux channel, without having 25 j =» . r ' 

j « . , j^T f v , , . . , ^ will be described below. As will be shown, these switches 

to send now control words to force the client device to slow . . A , . A - A . . 4 . 

, iL , 4 ,. - .„ , - » . . , iL are used in the context of the present invention to automati- 

down the rat at which it sends frames. In this scheme the «, A , - .. r . A . , - 4 . , 

i c^t- / » c r \ j i * ** j • . , t cally route around any failures m the links from the switch 

normal SOF (start of frame) delimiters used internal to the . ./ v , . v. 4 . . , . 4 

£1 _ • 1 . , ' ,. A j . « j i * j • • • ■ to the link card, or with the link card itself . 

fiber optic network have one bit modified to mdicate which „ . , , , . . , , 

link card channel the data is being sent to and from. 30 These switches also allowbackup paths to be provided by 

the fiber optic network that hop through these external 

In a second channel blending scheme, multip e link cards switches In Domal Uon mese bacfcu ^ ^ QOt be 

(in two more network nodes) are connected in a logicd I nng. ^ because me shortest routes ^ netwo ^ which 

All frames are encapsulated with a target link card ID. As do no , (hrQ h , hese swi(ch wju be ^ M me ^ 

frames amve in the MUX interface frame processor the daU tfas ^ ^ car<Js for ^ as backu (hs 

target hnk card ID is decoded. If the target hnk card ID in are marked in me fi es ^ hatched &a at . 

the frame matches the ID of the receiving hnk card, the tcms 

frame is stored in the appropriate outbound frame buffer. If rT ' , £1 _ , , , . „ T ^, - A iL 

. . , . ti~v » j ♦ , » .1 j * ■ f* j j However when a fiber breaks, as shown in FIG. 14, then 

the target link card ID s do not match, the data is forwarded 4 . , f4 , r , .„ , „ ' , . t , „ . , I , 

X j .1 u *u i^Tiv * r c me l oss of the link will be reflected to the attached switches 

to another node through the MUX interface frame processor. , c iL - , L1 , 4 4 . A 

»/. j t • , . . j l i .l l » a 40 and one or more of the switches is enabled to activate the 

If data is not currently being passed back through to the . n „ A . _ _ + M A , , A A 

**TTv-*f r_ j.f f.u backup path. As shown in FIG. 14, the switch at Customer 

MUX interface frame processor, data from one of the , . ... . , ^ , , , r . , . 

•u j c_ u <r .u u * UITV . node #2 is enabled, rou nng traffic back and forth between the 

inbound frame buffers can then be sent out to the MUX unit. . . . , A * A to , A , ^ r . 

rp * ji .i , A , . £ A _ . . two Imk cards at that node. Customer node #6 loses it s 

To handle the case when a pass through frame starts arriving A . « j j . A iL *■ *■ r.i_ 

<u .u univ ** u _i r* a. . • • • direct connection to the head end, but the activation of the 

from the MUX unit shortly after a frame transmission is . A , A _ A , ^- \. . , . 4 - 

*wjc„*u-u jf u & i ( r it 45 switch at Customer node #2 provides it with a path to the 

started from the inbound frame buffer, a supplemental buffer . , . . , r , — , A / 

is provided to buffer one frame inside the MUX interface ^ ^^f^ ^ \ , u ^T"! 

¥ j, v *u i- i . ,i * j * bandwidth of nodes 2 and 6 will be cut m half, though each 

frame processor. In addition, the link cards meter their data , t , . . . . - Ti f 

„ r iL . . K1 , , , 4tf . j node could use the original maximum bandwidth if the other 

flow onto the network by using a "leaky bucket" method- , . . Jt „ . &iu«u iuoauuuu, uouu iui U u ^ um & . 

i ♦ ir *u ■ u a m*u * -a a > ie node is idle. Customer node #3 loses one of it s direct paths 

ology to keep their bandwidth to a user specified amount. If t A , . . . , , . - A . A , , . 

• r i j ■ * 11 c ju j j*u 50 to the head end, but has a second path to the head end that 

a given hnk card is not using all of it s specified bandwidth, ^ ac( j ve 

it can send a bandwidth credit onto the network which „ * "• ^ „ 

another link card can claim to temporarily burst above its Referring to FIG. 15, a more complicated failover sce- 

user specified maximum. nan0 13 when one of me Mux cards m a customer node fails. 

In order to maximally use the bandwidth of the ring during 

Responding to Link Failures 55 norma l operation, nodes with just two link cards are con- 
figured to use the same Mux unit with one link card going 

The ring architecture of the present system can be used out one Mux port and the other hnk card going out the other 

with redundancy techniques to provide a complete redun- Mux port. If the Mux unit to which both hnk cards are 

dancy solution, enabling the system to be reconfigured in connected fails, and thereby cuts off both link cards from the 

response to virtually any component failure to as to restore 60 network, the link card controllers will configure both link 

data transmission services to all or almost all nodes of the cards to use a user specified backup path on the other Mux 

system. un it ^ th c external switch on that node will be activated, 

FIG. 12 shows the symbols that will be used in the as shown for Customer node #1 in FIG. 15. In a system 

following figures to represent one node of a fiber optic where all network data paths are used, this reconfiguration 

network in accordance with the present invention. Each Mux 65 will result in the two link cards of the node with the failed 

unit of the node is shown as a rectangular box, and each Mux unit being inserted in the middle of an already used 

optical fiber cable connected to the node is represented by link. 
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To other nodes on the fiber optic network, a client node 
Mux unit failure appears the same as a fiber break and thus 
is handled as discussed previously. 

Referring to FIG. 16, another failover mode is used when 
one of the "Head End" Mux units fails. To handle this 5 
correctly the Head End will be connected to the optical fiber 
cables so that each Mux unit is connected to both fiber optic 
rings. In addition, the external switches at two or more 
customer nodes will need to be activated. Thus if one of the 
Head End Mux units fails, each customer node will have at 10 
least one link card that is still connected by a live data path 
to the remaining alive Mux unit at the Head End. 

Alternate Embodiments 

While the present invention has been described with 35 
reference to a few specific embodiments, the description is 
illustrative of the invention and is not to be construed as 
limiting the invention. Various modifications may occur to 
those skilled in the art without departing from the true spirit 
and scope of the invention as defined by the appended 20 
claims. 

What is claimed is: 

1. Apparatus for use at a node of an optical fiber ring 
network, comprising: 

at least one link card that sends and receives a data stream 25 
to and from a client device; 

at least one multiplexer unit that sends and receives data 
streams to and from at least first and second optical 
fiber cables, each of the first and second optical fiber 3Q 
cables linking the node to another node; and 

switching apparatus for interconnecting the multiplexer 
units to the link cards; 

each multiplexer unit including an optical wavelength 
division multiplexer and demultiplexer that sends a first 35 
data stream over the first optical fiber cable and 
receives a second data stream from the first optical fiber 
cable. 

2. The apparatus of claim 1, wherein the optical wave- 
length division multiplexer and demultiplexer sends a third 40 
data stream over the second optical fiber cable and receives 

a fourth data stream over the second optical fiber cable. 

3. The apparatus of claim 2, wherein the optical wave- 
length division multiplexer and demultiplexer is a coarse 
optical wavelength division multiplexer and demultiplexer, 45 
which is operable to: 

send the first data stream using, and receive the fourth 
data stream at, a first optical wavelength, and 

send the second data stream using, and receive the third 
data stream at, a second optical wavelength. 50 

4. The apparatus of claim 1, wherein each link card 
includes an inbound FIFO frame buffer with memory capac- 
ity for storing at least 120 Fibre Channel frames sent by the 
client device to the link card and an outbound FIFO frame 
buffer with memory capacity for storing at least 120 Fibre 55 
Channel frames sent by another device to the link card for 
transmission to the client device. 

5. The apparatus of claim 4, wherein each link card 
includes circuitry that exchanges buffer credit signals with 
another link card coupled thereto by one of the first an 60 
second optical fiber cables so as to pre-fill the outbound 
FIFO frame buffer with frames of data before the client 
device sends flow control messages to request the transmis- 
sion of those data frames by another client device coupled to 
the other link card. 65 

6. The apparatus of claim 4, wherein each link card 
includes flow control circuitry for pre-filling the outbound 
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FIFO frame buffer with frames of data before the client 
device sends flow control messages to request the transmis- 
sion of those data frames. 

7. The apparatus of claim 1, wherein 

each link card includes an inbound FIFO frame buffer that 
stores frames sent by the client device to the link card 
and an outbound FIFO frame buffer that stores frames 
sent by another device to the link card for transmission 
to the client device; and 

the link card stores data into the inbound FIFO frame 
buffer and reads data from the outbound FIFO frame 
buffer at a first clock rate associated with the client 
device, and reads data from the inbound FIFO frame 
buffer for transmission to the multiplexer unit and 
stores data received from the multiplexer unit into the 
outbound FIFO frame buffer at a second clock rate 
associated with the fink card, whereby the data stream 
sent and received to and from the client device is 
retimed from the first clock rate to the second clock 
rate. 

8. The apparatus of claim 7, wherein the data stream sent 
and received to and from the client device is a Fibre Channel 
data stream, the first clock rate is approximately 1.0625 
Gbps, and the second clock rate is at least 1.25 Gbps. 

9. The apparatus of claim 7, wherein the multiplexer unit 
includes a smoothing circuit that retimes the data stream sent 
from the client device from the second clock rate to a third 
clock rate associated with the multiplexer unit, the multi- 
plexer unit transmitting the data stream over one of the first 
and second optical fiber cables at the third clock rate. 

10. Apparatus for use at a node of an optical fiber ring 
network, comprising: 

a first link card operable to receive a first data stream 
from, and send a second data stream to, a client device; 

a second link card operable to receive a third data stream 
from, and send a fourth data stream to, the client 
device; 

first and second multiplexer units each operable to merge 
the first and third data streams into a first combined data 
stream and transmit the first combined data stream over 
a first and second optical fiber cable, respectively, and 
each operable to receive a second combined data 
stream from the first and second optical fiber cable, 
respectively, and extract the second and fourth data 
streams from the second combined data stream; and 

switching apparatus for selectively connecting the first 
multiplexer unit with the first and second link cards; 

the first and second multiplexer units each including an 
optical wavelength division multiplexer and demulti- 
plexer that sends the first combined data stream over 
the first optical fiber cable and receives the second 
combined data stream from the first optical fiber cable. 

11. The apparatus of claim 10, wherein the first and 
second data streams for Fibre Channel data streams and the 
third and fourth data streams are Gigabit Ethernet data 
streams. 

12. The apparatus of claim 10, wherein at least one of the 
first link card and the first and second multiplexer units 
includes circuitry that inserts into the first data stream, prior 
to the first data stream being merged with the third data 
stream, marking symbols for marking the first data stream so 
as to enable a receiving device that receives the first com- 
bined data stream to identify the first data stream within the 
first combined data stream. 

13. The apparatus of claim 12, wherein the first and 
second multiplexer units each includes a demultiplexer that 
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demultiplexes the second combined data stream into the 
second and fourth data streams and that identifies the second 
data stream by identifying instances of the marking symbols 
in the second combined data stream. 

14. The apparatus of claim 10, wherein optical waveform 5 
division multiplexer and demultiplexer sends a third com- 
bined data stream over the second optical fiber cable and 
receives a fourth combined data stream over the second 
optical fiber cable. 

15. The apparatus of claim 14, wherein the optical wave- 10 
length division multiplexer and demultiplexer is a coarse 
optical wavelength division multiplexer and demultiplexer, 
which is operable to: 

send the first combined data stream using, and receive the 
fourth combined data stream at, a first optical * 5 
wavelength, and 

send the second data stream using, and receive the third 
data stream at, a second optical wavelength. 

16. The apparatus of claim 10, wherein the first and 
second link cards each include an inbound FIFO frame 20 
buffer with memory capacity for storing at least 120 Fibre 
Channel frames sent by the client device to the link card and 

an outbound FIFO frame buffer with memory capacity for 
storing at least 120 Fibre Channel frames sent by another 
device to the link card for transmission to the client device. 25 

17. The apparatus of claim 16, wherein the first and 
second link cards each include circuitry for exchanging 
buffer credit signals with another link card coupled thereto 
by one of the first and second optical fiber cables so as to 
pre-fill the outbound FIFO frame buffer with frames of data 30 
before the client device sends flow control messages to 
request the transmission of those data frames by another 
client device coupled to the other link card. 

18. The apparatus of claim 16, wherein the first and 
second link cards each includes flow control circuitry for 



139 Bl 

26 

pre-fill ing the outbound FIFO frame buffer with frames of 
data before the client device sends flow control messages to 
request the transmission of those data frames. 

19. The apparatus of claim 10, wherein 

the first and second link cards each include an inbound 
FIFO frame buffer for storing frames sent by the client 
device to the link card and an outbound FIFO frame 
buffer for storing frames sent by another device to the 
link card for transmission to the client device; and 

the first and second link cards each store data into the 
inbound FIFO frame buffer and read data from the 
outbound FIFO frame buffer at a first clock rate asso- 
ciated with the client device, and read data from the 
inbound FIFO frame buffer for transmission to at least 
one of the first and second multiplexer units and store 
data received from the multiplexer units into the out- 
bound FIFO frame buffer at a second clock rate asso- 
ciated with the link card, whereby the data stream sent 
and received to and from the client device is retimed 
from the first clock rate to the second clock rate. 

20. The apparatus of claim 19, wherein the first and 
second data streams sent and received to and from the client 
device are Fibre Channel data streams, the first clock rate is 
approximately 1.0625 Gbps, and the second clock rate is at 
least 1.25 Gbps. 

21. The apparatus of claim 19, wherein the first and 
second multiplexer units each includes a smoothing circuit 
that retimes the first data stream sent from the client device 
from the second clock rate to a third clock rate associated 
with the multiplexer unit, the first and second multiplexer 
units each transmitting the first data stream over one of the 
first and second optical fiber cables at the third clock rate. 
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